RQs or Hypotheses: RQ1: To what extent do video closed captions help students learn in fully online environments? RQ2: To what extent are student sub-groups differentially impacted by video closed caption use?
Variables: Learning (quiz grades, final assessment/exam); Learner Demographics (age/adult learner, first generation, income/pell grant eligibility, minority/ethnicity/race, disability, non-native English speakers)
Summary: A convenience population of 60 students in a 300-level biology course section were randomly assigned into one of two groups at the beginning of the term. At ten times during the term, each student group viewed a short video related to particular learning objectives for the course. Each group viewed each video in either the captioned condition (treatment) or the un-captioned condition (control). Students who viewed the captioned condition for the first video then viewed the un-captioned video for the second video, and vice-versa, alternating for all ten videos. Following each video, students were asked to take a short quiz and, at the end of the term, a final assessment of all of the learning objectives from all ten videos. Mann-Whitney U tests were conducted to determine the statistical significance of learning differences between the captioned and uncaptioned groups. No differences were found. Given the small sample, the study has continued through summer and fall sessions.
RQs or Hypotheses: H1: Students involved in the video production will have more successful learning and retention outcomes. H2: Student preparedness and readiness (online skills and self-directedness) will include student perceptions of engagement. H3: Student perceptions of engagement will influence student learning and performance.
Variables: Learning/Performance (perceptions, final exam, post-test, correct MC items, grade), Course Retention (completion/pass rate), Engagement, Satisfaction, Learner Online Skills, Learner Self-Directedness. Learner Demographics (Ethnicity, Income/Pell grant eligibility)
Summary: The process of creating the video involves deeper (higher-level) learning and peer efforts provide alternative ways of understanding a problem. The study used an experimental design based on random assignment of about 180 enrolled students across four online sections of two classes at the beginning of the Spring 2016 semester. Half of the sections included the video component, with common assessment instruments and other class activities across all classes. Variable data (demographics, engagement and satisfaction) came from student information systems, instructor gradebooks, and the DETA Research Toolkit Surveys administered on the class websites. Both student survey responses and data from course activity and performance were analyzed to test the hypotheses. The pilot study shows few significant differences, including across the two groups, with and without the intervention, although the performance of students in the upper-division class showed some significance in learning between groups based on post-test questions. Descriptive findings examining mean average on learning and retention were slightly higher for the treatment group than the control group. Few differences in these results emerge across the two groups of the student body enrolled in these classes.
According to structural equation modeling, learner online skills and self-directedness did not influence engagement. However, engagement did significantly impact the students’ perceived learning and performance (for both the treatment and control groups). This trend was strongest for Hispanics. Given the small sample, the study has continued through the fall session.
RQs or Hypotheses: H1: Open Educational Resources (OER), a free online book, will positively impact students’ performance and satisfaction.
Variables: Student performance (final course grade, day students started quiz, day student started coursework), Student Satisfaction.
Summary: This study sought to determine the impact upon student performance and satisfaction of replacing a high-cost textbook with a free, open access textbook in an Introduction to Psychology course at UW-Milwaukee. Through experimental design mode of course delivery, instructional design, instructor, grading scheme, assessments, academic term, and course content were held constant. The study found that students using an open textbook began their coursework earlier, made better progress across the term, needed fewer quiz attempts to earn their grade, and earned a higher mean final grade than those required to purchase a non-open textbook. Furthermore, no student expressed dissatisfaction with using the open textbook, and most felt it facilitated their learning.
RQs or Hypotheses: R1: Are student course activities and behaviors different in online courses than blended courses? R2: Do student course activities and behaviors influence student learning or satisfaction? R3: Do student perceptions of course assessment influence student learning or satisfaction?
Variables: Instructional Characteristics of Assessment, Students’ Online Behaviors (course activity, interactivity, and active learning), Learning, and Satisfaction. Summary: Through course announcements and with instructor permission, 2,949 (undergraduate and graduate) students from 15 online and 5 blended STEM courses were contacted to participate in the online survey, and a total of 1,767 complete and valid responses were collected.
Students in online courses received their content through reading, and in blended courses they received content through reviewing slides and taking notes. Statistically significant difference in these top course activities in which students participate is to recieve content. Also, there were significant differences in assessment. Students in blended courses reported taking more quizzes and exams, but students in online courses reported completing major projects and assignments. Descriptive statistics indicate that receiving content was the most frequent activity undertaken by students after taking quizzes/exams. Further, they indicated that interactivity most frequently tended to be communication from instructors via the news tool or email. There was seldom participation in discussions.
Student perceptions of learning and satisfaction can be predicted by student perceptions of course assessment and reported course activities and behaviors. Student perceptions of the course assessment, including assessment that is based on active learning, aligned with the outcomes, and with clearly established standards and methods of grading, can predict student learning and satisfaction with a course.
RQs or Hypotheses: H1: An online course’s instructional and course design (instructional characteristics), including learner support, organization and design, content, interactivity, assessment and evaluation, positively influence student outcomes (grades, completion, perception of performance, learning, and satisfaction).
Variables: Instructional Characteristics/Course Design (learner support, organization and design, content, interactivity, assessment and evaluation), Student Performance (grades), Course Attrition (completion), Learning, Satisfaction Summary: The student survey instrument included five measures of instructional characteristics, several measures of student outcomes, including learning, satisfaction, and performance, in addition to key demographic characteristics, was distributed by email to randomly chosen students enrolled in at least one online course during the Spring 2016 semester. Data collection ran for one month and was followed with Student Information Services (SIS) data acquisition from each institution’s internal data departments. Information from SIS was merged with student survey data. For the instructional characteristics study, the final sample size was 131 students.
Using hierarchical regression and five measures of instructional characteristics, this study found that the design and organization component of an online course, separately, is important instructional characteristics significantly predicting student outcomes. Design and organization was found to positively impact student learning and satisfaction after controlling for known covariates influencing student outcomes. Design and organization includes alignment with learning objectives, types of learning objectives, and overall organization of the course. Furthermore, the instructional characteristics as a whole, including all five components learner support, organization and design, content, interactivity, and assessment and evaluation, positively influence student learning and satisfaction after controlling for known covariates influencing student outcomes. Instructional characteristics did not predict final grade.
RQs or Hypotheses: R1: Does the use of active learning strategies represent a technique applicable to distance learning and retain the same level of effectiveness when compared to face-to-face instruction?
Summary: Active learning strategies involve a set of instructional practices: (a) problem-based instruction where students work to solve challenges, (b) collaborative exercises among peers to fulfill assignments, (c) expectations involving student participation in defining and responding to tasks, and (d) learning platforms that require active student engagement for progression. An extensive search of electronic literature databases as well as the reference sections of existing articles were searched. More than 1,000 manuscripts were ultimately collected and examined for possible inclusion. For inclusion, a study has to examine a course entirely taught online that incorporated some element of active learning. It had to either made a comparison to: (a) a distance learning control group that did not use active learning strategies, (b) measure change in achievement over time longitudinally, or (c) make a comparison to a face-to-face learning environment. The results indicate that active learning strategies, adapted for distance education, demonstrate the ability to improve educational outcomes. The results indicate that incorporating active learning improved test scores when compared to control groups that did not incorporate active learning. The size of the learning compares favorably to that expected in a face-to-face setting using an equivalent active learning approach.
RQs or Hypotheses: R1: What do student persistence patterns look like in comptency-based education?
Summary: Examining a non-term, direct assessment competency-based program, student enrollment and grade patterns for 569 students who enrolled during the first two calendar years of the program we analuzed. Findings indicate that students who mastered at least one course-equivalent during their initial enrollment in the program were retained and persisted at higher rates than those who did not, regardless of the other grades they received. Mapping patterns of student grades and retention into later enrollments found that students who mastered at least one course-equivalent during their initial enrollment were far more likely to be retained in the program and persist after their second enrollment regardless of that enrollment’s outcome.
DETA Research Questions:
What are course components (content, interactivity, and assessment) that influence student outcomes?
What patterns of behaviors lead to increased student outcomes?
Study 1: Content: Experimental
Intervention: Closed Captioning
RQs or Hypotheses: RQ1: To what extent do video closed captions help students learn in fully online environments? RQ2: To what extent are student sub-groups differentially impacted by video closed caption use?
Variables: Learning (quiz grades, final assessment/exam); Learner Demographics (age/adult learner, first generation, income/pell grant eligibility, minority/ethnicity/race, disability, non-native English speakers)
Summary: A convenience population of 60 students in a 300-level biology course section were randomly assigned into one of two groups at the beginning of the term. At ten times during the term, each student group viewed a short video related to particular learning objectives for the course. Each group viewed each video in either the captioned condition (treatment) or the un-captioned condition (control). Students who viewed the captioned condition for the first video then viewed the un-captioned video for the second video, and vice-versa, alternating for all ten videos. Following each video, students were asked to take a short quiz and, at the end of the term, a final assessment of all of the learning objectives from all ten videos. Mann-Whitney U tests were conducted to determine the statistical significance of learning differences between the captioned and uncaptioned groups. No differences were found. Given the small sample, the study has continued through summer and fall sessions.
Study 2: Content: Experimental
Intervention: Student-produced Video
RQs or Hypotheses: H1: Students involved in the video production will have more successful learning and retention outcomes. H2: Student preparedness and readiness (online skills and self-directedness) will include student perceptions of engagement. H3: Student perceptions of engagement will influence student learning and performance.
Variables: Learning/Performance (perceptions, final exam, post-test, correct MC items, grade), Course Retention (completion/pass rate), Engagement, Satisfaction, Learner Online Skills, Learner Self-Directedness. Learner Demographics (Ethnicity, Income/Pell grant eligibility)
Summary: The process of creating the video involves deeper (higher-level) learning and peer efforts provide alternative ways of understanding a problem. The study used an experimental design based on random assignment of about 180 enrolled students across four online sections of two classes at the beginning of the Spring 2016 semester. Half of the sections included the video component, with common assessment instruments and other class activities across all classes. Variable data (demographics, engagement and satisfaction) came from student information systems, instructor gradebooks, and the DETA Research Toolkit Surveys administered on the class websites. Both student survey responses and data from course activity and performance were analyzed to test the hypotheses.
The pilot study shows few significant differences, including across the two groups, with and without the intervention, although the performance of students in the upper-division class showed some significance in learning between groups based on post-test questions. Descriptive findings examining mean average on learning and retention were slightly higher for the treatment group than the control group. Few differences in these results emerge across the two groups of the student body enrolled in these classes.
According to structural equation modeling, learner online skills and self-directedness did not influence engagement. However, engagement did significantly impact the students’ perceived learning and performance (for both the treatment and control groups). This trend was strongest for Hispanics. Given the small sample, the study has continued through the fall session.
Study 3: Content: Experimental
Intervention: OER
RQs or Hypotheses: H1: Open Educational Resources (OER), a free online book, will positively impact students’ performance and satisfaction.
Variables: Student performance (final course grade, day students started quiz, day student started coursework), Student Satisfaction.
Summary: This study sought to determine the impact upon student performance and satisfaction of replacing a high-cost textbook with a free, open access textbook in an Introduction to Psychology course at UW-Milwaukee. Through experimental design mode of course delivery, instructional design, instructor, grading scheme, assessments, academic term, and course content were held constant. The study found that students using an open textbook began their coursework earlier, made better progress across the term, needed fewer quiz attempts to earn their grade, and earned a higher mean final grade than those required to purchase a non-open textbook. Furthermore, no student expressed dissatisfaction with using the open textbook, and most felt it facilitated their learning.
Study 4: Assessment and Student Behaviors: Observational
Intervention: Observational study
RQs or Hypotheses: R1: Are student course activities and behaviors different in online courses than blended courses? R2: Do student course activities and behaviors influence student learning or satisfaction? R3: Do student perceptions of course assessment influence student learning or satisfaction?
Variables: Instructional Characteristics of Assessment, Students’ Online Behaviors (course activity, interactivity, and active learning), Learning, and Satisfaction.
Summary: Through course announcements and with instructor permission, 2,949 (undergraduate and graduate) students from 15 online and 5 blended STEM courses were contacted to participate in the online survey, and a total of 1,767 complete and valid responses were collected.
Students in online courses received their content through reading, and in blended courses they received content through reviewing slides and taking notes. Statistically significant difference in these top course activities in which students participate is to recieve content. Also, there were significant differences in assessment. Students in blended courses reported taking more quizzes and exams, but students in online courses reported completing major projects and assignments. Descriptive statistics indicate that receiving content was the most frequent activity undertaken by students after taking quizzes/exams. Further, they indicated that interactivity most frequently tended to be communication from instructors via the news tool or email. There was seldom participation in discussions.
Student perceptions of learning and satisfaction can be predicted by student perceptions of course assessment and reported course activities and behaviors. Student perceptions of the course assessment, including assessment that is based on active learning, aligned with the outcomes, and with clearly established standards and methods of grading, can predict student learning and satisfaction with a course.
Study 5: Content/Interactivity/Assessment: Observational study
RQs or Hypotheses: H1: An online course’s instructional and course design (instructional characteristics), including learner support, organization and design, content, interactivity, assessment and evaluation, positively influence student outcomes (grades, completion, perception of performance, learning, and satisfaction).
Variables: Instructional Characteristics/Course Design (learner support, organization and design, content, interactivity, assessment and evaluation), Student Performance (grades), Course Attrition (completion), Learning, Satisfaction
Summary: The student survey instrument included five measures of instructional characteristics, several measures of student outcomes, including learning, satisfaction, and performance, in addition to key demographic characteristics, was distributed by email to randomly chosen students enrolled in at least one online course during the Spring 2016 semester. Data collection ran for one month and was followed with Student Information Services (SIS) data acquisition from each institution’s internal data departments. Information from SIS was merged with student survey data. For the instructional characteristics study, the final sample size was 131 students.
Using hierarchical regression and five measures of instructional characteristics, this study found that the design and organization component of an online course, separately, is important instructional characteristics significantly predicting student outcomes. Design and organization was found to positively impact student learning and satisfaction after controlling for known covariates influencing student outcomes. Design and organization includes alignment with learning objectives, types of learning objectives, and overall organization of the course. Furthermore, the instructional characteristics as a whole, including all five components learner support, organization and design, content, interactivity, and assessment and evaluation, positively influence student learning and satisfaction after controlling for known covariates influencing student outcomes. Instructional characteristics did not predict final grade.
Study 6: Interactivity: Meta-analysis
RQs or Hypotheses: R1: Does the use of active learning strategies represent a technique applicable to distance learning and retain the same level of effectiveness when compared to face-to-face instruction?
Summary: Active learning strategies involve a set of instructional practices: (a) problem-based instruction where students work to solve challenges, (b) collaborative exercises among peers to fulfill assignments, (c) expectations involving student participation in defining and responding to tasks, and (d) learning platforms that require active student engagement for progression. An extensive search of electronic literature databases as well as the reference sections of existing articles were searched. More than 1,000 manuscripts were ultimately collected and examined for possible inclusion. For inclusion, a study has to examine a course entirely taught online that incorporated some element of active learning. It had to either made a comparison to: (a) a distance learning control group that did not use active learning strategies, (b) measure change in achievement over time longitudinally, or (c) make a comparison to a face-to-face learning environment. The results indicate that active learning strategies, adapted for distance education, demonstrate the ability to improve educational outcomes. The results indicate that incorporating active learning improved test scores when compared to control groups that did not incorporate active learning. The size of the learning compares favorably to that expected in a face-to-face setting using an equivalent active learning approach.
Study 7: Competency-based Education (CBE) and Student Success
RQs or Hypotheses: R1: What do student persistence patterns look like in comptency-based education?
Summary: Examining a non-term, direct assessment competency-based program, student enrollment and grade patterns for 569 students who enrolled during the first two calendar years of the program we analuzed. Findings indicate that students who mastered at least one course-equivalent during their initial enrollment in the program were retained and persisted at higher rates than those who did not, regardless of the other grades they received. Mapping patterns of student grades and retention into later enrollments found that students who mastered at least one course-equivalent during their initial enrollment were far more likely to be retained in the program and persist after their second enrollment regardless of that enrollment’s outcome.